Focus control device, endoscope apparatus, and method for operating focus control device

ABSTRACT

A focus control device includes a processor including hardware. The processor sets a plurality of regions, each including a plurality of pixels, to an image acquired by an imaging section, obtains a direction determination result for each region in some or all of the plurality of regions set, by determining whether a target focusing position that is a target of an in-focus object plane position is on a NEAR side or a FAR side relative to a reference position, determines an in-focus direction by performing weighted comparison between NEAR area information and FAR area information, based on the direction determination result and weight information, and controls the in-focus object plane position based on the in-focus direction.

CROSS REFERENCE TO RELATED APPLICATION

This application is a continuation of International Patent ApplicationNo. PCT/JP2016/051138, having an international filing date of Jan. 15,2016, which designated the United States, the entirety of which isincorporated herein by reference.

BACKGROUND

In recent years, the depth of field of an endoscope system has becomeshallow along with the use of an image sensor having a large number ofpixels. In view of this, an endoscope system that performs an autofocus(AF) process has been proposed.

An endoscopic procedure involves ablating a lesioned part, suturing, andthe like. Thus, a treatment tool such as an electrocauter and forcepsmight be disposed between a tissue that is a focusing target and anendoscope system serving as an imaging device. In such a case, atreatment tool that has a higher contrast than a tissue might be broughtinto focus, and thus the tissue might fail to be brought into focus.

JP-A-2006-245792 discloses a technique to address this. Specifically, ina case where there is an obstacle between the target subject and animaging device, the target subject is brought into focus with theobstacle designated by a user.

SUMMARY

According to one aspect of the invention, there is provided a focuscontrol device comprising a processor including hardware,

the processor being configured to implement:

a region setting process of setting a plurality of regions, eachincluding a plurality of pixels, to an image acquired by an imagingsection;

a direction determination process of obtaining a direction determinationresult for each region in some or all of the plurality of regions set,by determining whether a target focusing position that is a target of anin-focus object plane position is on a NEAR side or a FAR side relativeto a reference position; and

focus control of determining an in-focus direction by performingweighted comparison between NEAR area information, indicating an area ofa region with the determination result NEAR, and FAR area information,indicating an area of a region with the determination result FAR, basedon weight information and the direction determination result, and ofcontrolling the in-focus object plane position based on the in-focusdirection.

According to another aspect of the invention, there is provided anendoscope apparatus comprising the above focus control device.

According to another aspect of the invention, there is provided a methodfor operating a focus control device, the method comprising:

setting a plurality of regions, each including a plurality of pixels, toan image acquired by an imaging section;

performing a direction determination process of obtaining a directiondetermination result for each region in some or all of the plurality ofregions set, by determining whether a target focusing position that is atarget of an in-focus object plane position is on a NEAR side or a FARside relative to a reference position; and

determining an in-focus direction by performing weighted comparisonbetween NEAR area information, indicating an area of a region with thedetermination result NEAR, and FAR area information, indicating an areaof a region with the determination result FAR, based on weightinformation and the direction determination result, and controlling thein-focus object plane position based on the in-focus direction thusdetermined.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 illustrates an example of positional relationship between anendoscope apparatus (imaging section) and subjects (tissue, treatmenttool).

FIG. 2 illustrates a basic configuration example of a focus controldevice.

FIG. 3 illustrates a configuration example of the endoscope apparatusincluding the focus control device.

FIG. 4 illustrates a configuration example of an AF control section.

FIG. 5 is a flowchart illustrating focus control.

FIG. 6 is a flowchart illustrating a focusing operation.

FIG. 7 illustrates an example of how a plurality of regions (evaluationblocks) are set.

FIG. 8A is a diagram illustrating an example of how a focus lensaccording to an embodiment is controlled in a direction determinationprocess, and FIG. 8B is a diagram illustrating a conventional method.

FIG. 9 is a flowchart illustrating a block state determination process.

FIG. 10 illustrates a specific example of how a block state changes overtime.

FIG. 11 illustrates an invalid frame setting process based on adirection determination result.

FIG. 12 is a flowchart illustrating an in-focus direction determinationprocess.

FIG. 13 illustrates an example of how an in-focus position is adjustedby adjusting second weight information (threshold).

FIG. 14 is a diagram illustrating a target distance estimation process.

FIG. 15 is a diagram illustrating a target region detection process.

FIG. 16 is another flowchart illustrating the in-focus directiondetermination process.

FIG. 17 is a diagram illustrating how the focus lens is controlled in anin-focus determination process.

FIG. 18 is a flowchart illustrating the in-focus determination process.

FIG. 19 is a flowchart illustrating a focus lens position determinationprocess.

FIG. 20 illustrates a specific example of how the block state changesover time.

DESCRIPTION OF EXEMPLARY EMBODIMENTS

According to one embodiment of the invention, there is provided a focuscontrol device comprising a processor including hardware,

the processor being configured to implement:

a region setting process of setting a plurality of regions, eachincluding a plurality of pixels, to an image acquired by an imagingsection;

a direction determination process of obtaining a direction determinationresult for each region in some or all of the plurality of regions set,by determining whether a target focusing position that is a target of anin-focus object plane position is on a NEAR side or a FAR side relativeto a reference position; and

focus control of determining an in-focus direction by performingweighted comparison between NEAR area information, indicating an area ofa region with the determination result NEAR, and FAR area information,indicating an area of a region with the determination result FAR, basedon weight information and the direction determination result, and ofcontrolling the in-focus object plane position based on the in-focusdirection.

According to another embodiment of the invention, there is provided anendoscope apparatus comprising the above focus control device.

According to another embodiment of the invention, there is provided amethod for operating a focus control device, the method comprising:

setting a plurality of regions, each including a plurality of pixels, toan image acquired by an imaging section;

performing a direction determination process of obtaining a directiondetermination result for each region in some or all of the plurality ofregions set, by determining whether a target focusing position that is atarget of an in-focus object plane position is on a NEAR side or a FARside relative to a reference position; and

determining an in-focus direction by performing weighted comparisonbetween NEAR area information, indicating an area of a region with thedetermination result NEAR, and FAR area information, indicating an areaof a region with the determination result FAR, based on weightinformation and the direction determination result, and controlling thein-focus object plane position based on the in-focus direction thusdetermined.

Exemplary embodiments of the invention are described below. Note thatthe following exemplary embodiments do not in any way limit the scope ofthe invention laid out in the claims. Note also that not all of theelements described below in connection with the exemplary embodimentsshould be taken as essential elements of the invention.

1. Method According to Present Embodiment

First of all, a method according to the present embodiment is described.A subject other than a target subject of a user might be included in acaptured image to be an obstacle. In such a case, the target subject ispreferably in a state of being easily monitorable in the captured image,that is, a state of being in focus. Unfortunately, the target subjectmight not necessarily brought into focus by simply employing autofocus(AF). An endoscopic procedure involves ablating a lesioned part,suturing, and the like. Thus, as illustrated in FIG. 1, a treatment toolsuch as an electrocauter and forceps might be disposed between a tissuethat is a focusing target and an endoscope apparatus (endoscope system)serving as an imaging device. Contrast AF, with which a region with ahigher contrast is brought into focus, might result in a treatment toolbrought into focus instead of the tissue that is supposed to be broughtinto focus.

In this context, a desired subject can be accurately brought into focuswith a method described in JP-A-2006-245792, featuring designation, madeby the user, of a subject serving as an obstacle. However, a status ofan obstacle in the captured image might frequently change under apredetermined circumstance. In such a case, a large operation load isimposed on the user required to designate the obstacle each time thechange occurs.

For example, during an endoscopic procedure, such as a laparoscopicsurgery, a treatment tool is inserted into the body together with ascope (imaging section), and the treatment is performed on a tissueusing the treatment tool. This treatment tool is a tool used for atreatment for a tissue. Specific examples of the treatment tool includean energy device (such as an electrocauter), forceps, and the like. Thetreatment tool is used for a treatment for a tissue, and thus isfrequently moved by a user (a doctor, a physician). For example, amembranous tissue is pulled up using forceps, or a tissue immobilizedwith forceps is ablated using an electrocauter. As a result, the sizeand the position of the treatment tool in the captured image frequentlychange. Thus, a region of the obstacle in the captured image frequentlychanges, rendering the manual designation by the user extremelycumbersome.

In view of this, the system may perform a process of lowering the levelof contribution of the region of the obstacle in the captured image tothe AF control (process of excluding the region from the AF control in anarrow sense), whereby the target subject can be appropriately broughtinto focus.

In view of this, the present applicant proposes the following focuscontrol device. As illustrated in FIG. 2, the focus control deviceaccording to the present embodiment includes a region setting section2010, a direction determination section 2040, and a focus controlsection 2000. The region setting section 2010 sets a plurality ofregions (evaluation blocks), each including a plurality of pixels, to animage acquired by an imaging section (corresponding to an imagingsection 200 illustrated in FIG. 3 described later). The directiondetermination section 2040 implements a direction determination processof obtaining a direction determination result for each region in some orall of the plurality of regions set, by determining whether a targetfocusing position that is a target of the in-focus object plane positionis on a NEAR side or a FAR side relative to a reference position. Thefocus control section 2000 determines the in-focus direction byperforming weighted comparison between NEAR area information, indicatingan area of a region with a determination result NEAR, and FAR areainformation, indicating an area of a region with a determination resultFAR, based on the direction determination result and weight information.Then, the focus control section 2000 controls the in-focus object planeposition based on the in-focus direction thus determined.

The in-focus object plane position as used herein represents a positionof an object when a system, including an optical system (an objectivelens system 240 in FIG. 3 described later in a narrow sense), an imageplane (a plane of an image sensor 250 in FIG. 3), and an object(subject), is in an in-focus state. For example, as illustrated in FIG.3, which will be referred to later, in a case where the image sensor 250is fixed and a focus lens 220 in the optical system is movable, thein-focus object plane position is determined by determining the positionof the focus lens 220. In this case, an image in which a subjectpositioned within a range of depth of field including the in-focusobject plane position is focused is acquired.

Note that NEAR and FAR each indicate the direction of the targetfocusing position relative to the reference position. The result NEAR isobtained when the target focusing position is closer to the imagingsection 200 (the optical system and the image sensor 250) than thereference position is. The result FAR is obtained when the targetfocusing position is farther from the imaging section 200 than thereference position is. When the in-focus object plane position can becontrolled based on the position of the focus lens 220 as in the exampleillustrated in FIG. 3, the in-focus object plane position can becontrolled to move toward the NEAR side with the position of the focuslens 220 moved toward the near side, and can be controlled to movetoward the FAR side with the position of the focus lens 220 moved towardthe far side.

The weighted comparison is a process of comparing the NEAR areainformation and the FAR area information after weighting using theweight information. The weight information is at least one of firstweight information indicating a level of contribution of each of aplurality of regions to the weighted comparison and second weightinformation indicating a level of contribution of the NEAR areainformation and the FAR area information to the weighted comparison. Inother words, the weight information according to the present embodimentmay be a weight used for calculating the NEAR area information (or theFAR area information or both) or may be a weight of the NEAR areainformation (or the FAR area information or both) itself.

With this configuration, the target subject can be brought into focuswithout requiring the user to perform a cumbersome operation. Inparticular, the configuration can provide an endoscope apparatus havingan AF control function with which the target subject can be brought intofocus without requiring the user to perform a cumbersome operation invarious possible scenes during the endoscopic procedure. Specifically,appropriate AF control can be achieved due to the following threepoints.

1. In the present embodiment, the direction of moving the in-focusobject plane position is determined based on the area information. Thearea information is determined based on the direction determinationresult (NEAR or FAR). Once the direction determination result isacquired based on the AF evaluation value (contrast value), themagnitude of the AF evaluation value of each region contributes to noneof the processes thereafter. Generally, the treatment tool has a highercontrast than tissues. Thus, in normal contrast AF, a region including atreatment tool that has been failed to be detected is likely to have alarge impact when used in the AF control. In the method according to thepresent embodiment, the in-focus direction is determined by areainformation comparison (weighted comparison). Thus, each region(evaluation block) is only worth a single vote regardless of the AFevaluation value of the subject, whereby the impact of the treatmenttool can be limited.

2. In the present embodiment, the weight of each region for obtainingthe area information (the weight of each region with a determinationresult NEAR for obtaining the NEAR area information in a narrow sense)can be set by using the first weight information. Thus, as describedlater with reference to FIG. 15, when a target region in the image canbe identified, the target region can be brought into focus withpriority.

3. In the present embodiment, the second weight information is used sothat the NEAR area information and the FAR area information are notsimply compared but are weighted to be compared. For example, NEAR areainformation S_(N) and FAR area information S_(F) can be compared todetermine whether or not the following Formula (1) is satisfied insteadof determining whether or not S_(N)>S_(F) is satisfied.

M×S _(N) >S _(F)  (1),

where M represents the second weight information.

This Formula (1) can be converted into the following Formula (2) with Msubstituted with (1−Th)/Th,

S _(N)/(S _(N) +S _(F))>Th  (2).

The left side in this Formula (2) corresponds to a ratio nearRatio of aNEAR block described later, and thus the weight information (secondweight information) according to the present embodiment may be thresholdTH_NEAR for determining the in-focus direction. As described later withreference to FIG. 13, the threshold TH_NEAR serves as a parameter foradjusting an in-focus position (the in-focus object plane position atthe point when the focusing operation is completed). Thus, with thesecond weight information, the in-focus position can be flexiblyadjusted. For example, a value of the second weight information may beadjusted based on a result of estimating a distance to the targetsubject, as described later with reference to FIG. 14.

The focus control device according to the present embodiment includes amemory (storage section) that stores therein information (for example, aprogram and various types of data) and a processor (a processing section300 in FIG. 3, a processor including hardware) that operates based onthe information stored in the memory. The processor performs a regionsetting process of setting a plurality of regions, each including aplurality of pixels, to an image acquired by an imaging section, adirection determination process of obtaining a direction determinationresult for each region in some or all of the plurality of regions set,by determining whether a target focusing position that is a target of anin-focus object plane position is on a NEAR side or a FAR side relativeto a reference position, and focus control of determining an in-focusdirection by performing weighted comparison between NEAR areainformation, indicating an area of a region with the determinationresult NEAR, and FAR area information, indicating an area of a regionwith the determination result FAR, based on the direction determinationresult and weight information, and controlling the in-focus object planeposition based on the in-focus direction thus determined.

For example, the processor may have functions of sections eachimplemented by individual hardware, or the functions of sectionsimplemented by integrated hardware. The processor may be a centralprocessing unit (CPU), for example. Note that the processor is notlimited to the CPU. Various other processors such as a graphicsprocessing unit (GPU) or a digital signal processor (DSP) may also beused. The processor may be a hardware circuit that includes anapplication specific integrated circuit (ASIC). The memory may be asemiconductor memory (e.g., SRAM or DRAM), or may be a register. Thememory may be a magnetic storage device such as a hard disk drive (HDD),or may be an optical storage device such as an optical disc device, forexample. For example, the memory stores a computer-readable instruction,and the function of each section of the focus control device isimplemented by causing the processor to perform the instruction. Theinstruction may be an instruction set that is included in a program, ormay be an instruction that instructs the hardware circuit included inthe processor to operate.

An operation according to the present embodiment is implemented asfollows for example. The processor performs a process of setting aplurality of regions to an acquired image, and stores information on theplurality of regions in the memory. The processor reads the informationon the plurality of regions from the memory, obtains the AF evaluationvalue (block AF evaluation value) of each of the regions, and stores thevalues in the memory. The processor reads the block AF evaluation valuefrom the memory, performs the direction determination process, andstores a direction determination result in the memory. The processorreads the direction determination result and the weight information fromthe memory, performs the weighted comparison to obtain the in-focusdirection, and stores the direction in the memory. The processor readsthe in-focus direction from the memory, and controls the in-focus objectplane position based on the in-focus direction. Specifically, thein-focus object plane position may be controlled by a process ofoutputting a control signal to a mechanism (a focus lens driving section230 in FIG. 3) that drives the focus lens.

The sections of the focus control device according to the presentembodiment are implemented as modules of a program operating on aprocessor. For example, the region setting section is implemented as aregion setting module that sets a plurality of regions, each including aplurality of pixels, to an image acquired by the imaging section. Thedirection determination section is implemented as a directiondetermination module that implements a direction determination processof obtaining a direction determination result for each region in some orall of the plurality of regions set, by determining whether a targetfocusing position that is a target of the in-focus object plane positionis on a NEAR side or a FAR side relative to a reference position. Thefocus control section is implemented as a focus control module thatdetermines the in-focus direction by performing weighted comparisonbetween NEAR area information, indicating an area of a region with adetermination result NEAR, and FAR area information, indicating an areaof a region with a determination result FAR, based on the directiondetermination result and weight information, and controls the in-focusobject plane position based on the in-focus direction thus determined.

The present embodiment is described in detail below. First of all, asystem configuration example of a focus control device according to thepresent embodiment and an endoscope apparatus including the focuscontrol device is described. Then, an overview of the focusing operationaccording to the present embodiment is described. An apparatus(electronic apparatus) including the focus control device according tothe present embodiment is not limited to the endoscope apparatus and maybe any other apparatus. For example, an apparatus such as a digitalstill camera, a video camera, or a mobile phone may include the focuscontrol device according to the present embodiment. Also in such a case,flexible AF control can be achieved with the impact of an obstaclelimited and without requiring the user to go through a cumbersomeoperation.

2. System Configuration Example

The endoscope apparatus including the focus control device according tothe present embodiment is described with reference to FIG. 3. Theendoscope apparatus according to the present embodiment includes a rigidscope 100 that is inserted into a body, the imaging section 200 that isconnected to the rigid scope 100, the processing section 300, a displaysection 400, an external I/F section 500, and a light source section600.

The light source section 600 includes a white light source 610 thatemits white light, and a light guide cable 620 that guides the lightemitted from the white light source 610 to the rigid scope. The rigidscope 100 includes a lens system 110 that includes an imaging lens, arelay lens, an eyepiece, and the like, and a light guide section 120that guides the light emitted from the light guide cable 620 to the endof the rigid scope. The imaging section 200 includes the objective lenssystem 240 that forms an image of the light emitted from the lens system110. The objective lens system 240 includes the focus lens 220 thatadjusts the in-focus object plane position. The imaging section 200 alsoincludes the image sensor 250 that photoelectrically converts thereflected light focused by the objective lens system 240 to generate animage, the focus lens driving section 230 that drives the focus lens220, and an autofocus (AF) start/stop button 210 that controls AFstart/stop. The focus lens driving section 230 is a voice coil motor(VCM), for example.

The image sensor 250 has a structure in which a plurality of pixels arearranged in a two-dimensional array, and R, G, and B color filters aredisposed in a Bayer array on a pixel basis. Alternatively, an imagesensor that utilizes a complementary color filter, a stacked imagesensor that is designed so that each pixel can receive light having adifferent wavelength without using a color filter, and a monochromeimage sensor that does not utilize a color filter may be employed aslong as the subject can be captured to obtain an image.

The processing section 300 includes an A/D conversion section 310, apre-processing section 320, an image processing section 330, an AFcontrol section 340, and a control section 350. The A/D conversionsection 310 converts analog signals sequentially output from the imagesensor 250 into digital images and sequentially outputs the digitalimages to the pre-processing section 320. The pre-processing section 320performs image processing including white balance, an interpolationprocess (demosaicing process), and the like on the images output fromthe A/D conversion section 310, and sequentially outputs the resultantimages to the image processing section 330 and the AF control section340. The details of the AF control section 340 are described later. Theimage processing section 330 performs image processing including colorconversion, gray scale conversion, edge enhancement, a scaling process,a noise reduction, and the like on the images output from thepre-processing section 320, and sequentially outputs the resultantimages to the display section 400. The display section 400 is a liquidcrystal monitor for example, and displays the image sequentially outputfrom the image processing section 330.

The control section 350 is connected to the external I/F section 500,the image processing section 330, the AF control section 340, the imagesensor 250, the AF start/stop button 210, and the like to exchange acontrol signal. The external I/F section 500 is an interface that allowsthe user to perform an input operation on the endoscope apparatus, forexample. For example, the external I/F section 500 includes a modebutton for switching the AF mode, a setting button for setting theposition and the size of the AF area, an adjustment button for adjustingimage processing parameters, and the like.

As illustrated in FIG. 4, the AF control section 340 includes, forexample, the region setting section 2010, a block feature amountcalculation section 2020, an AF evaluation value calculation section2030, the direction determination section (block direction determinationsection) 2040, an invalid block setting section (invalid region settingsection) 2050, a block state determination section 2060, an invalidframe setting section 2070, an in-focus direction determination section2080, an in-focus determination section 2090, and a focus lens positiondetermination section 2095.

The region setting section 2010 sets regions, used for the AF, to thecaptured image. The regions may include both AF regions and evaluationblocks. The block feature amount calculation section 2020 calculates afeature amount for each evaluation block. The AF evaluation valuecalculation section 2030 calculates an evaluation value, used for theAF, for each evaluation block. The direction determination section 2040determines the direction toward the target focusing position based onthe AF evaluation value for each evaluation block. This directiondetermination result is information indicating NEAR or FAR in a narrowsense. The invalid block setting section 2050 sets an invalid blockbased on the feature amount. The invalid block is an evaluation blocknot used in the in-focus direction determination. The block statedetermination section 2060 determines a block state that is a finaldirection determination result, based on history information on thedirection determination result. The invalid frame setting section 2070determines whether or not a process target frame is to be set as aninvalid frame. The invalid frame is a frame not used in the in-focusdirection determination. The in-focus direction determination section2080 determines the in-focus direction, that is, a movement direction ofthe in-focus object plane position (or the movement direction of thefocus lens 220 corresponding to the movement direction). The in-focusdetermination section 2090 determines whether or not the in-focus stateis achieved by the movement of the in-focus object plane position, thatis, whether or not to terminate the focusing operation. The focus lensposition determination section 2095 determines a position to which thefocus lens is moved. Specifically, the position is determined based onthe movement in the in-focus direction obtained (movement of thewobbling center position) and the movement (wobbling operation) fordetermining the direction.

Processes performed by the sections of the AF control section 340 aredescribed in detail later. For example, the focus control section 2000in FIG. 2 may correspond to the components of the AF control section 340illustrated in FIG. 4 excluding the region setting section 2010 and thedirection determination section 2040. The focus control device accordingto the present embodiment may correspond to the AF control section 340.However, this should not be construed in a limiting sense, and variousmodifications may be made including a modification where the entireprocessing section 300 in FIG. 3 serves as the focus control device. Thefocus control device may be modified in various ways with the componentspartially omitted, or additional components provided. Variousmodifications may be made also on other configurations in FIG. 3 andFIG. 4.

3. Detail of Process Performed by AF Control Section

<Overview>

An overview of the AF control performed by the AF control section 340according to the present embodiment is described with reference to FIG.5. When the AF starts as a result of the user operating the AFstart/stop button 210, the AF control section 340 starts the focusingoperation.

When the focusing operation starts, the wobbling operation of the focuslens starts in synchronization with acquisition timings of imagessequentially output from the A/D conversion section 310, and thefocusing operation is performed based on the images acquired during thewobbling operation (S101).

Then, whether or not the focusing operation has been successfullycompleted with the subject brought into focus is determined (S102). Whenthe focusing operation has not been completed, the focus lens positionis changed (S103) and the focusing operation is performed again. Thefocusing operation in S100 is performed for each frame for example.Thus, the focusing operation is performed and whether or not thefocusing operation has been completed is determined for each frame. Theoperation is performed until the focusing operation is determined tohave been completed, while changing the focus lens position. When thefocusing operation is completed in S102, the focus lens is moved to thein-focus position, and the focusing operation (wobbling) is terminated(S104).

When the focusing operation is terminated, the AF control section 340starts the standby operation. When the standby operation starts, the AFcontrol section 340 performs a scene change detection process (S105).For example, the AF control section 340 uses images sequentially outputfrom the pre-processing section 320 to detect a scene change bymonitoring changes in the color, the luminance, or the AF evaluationvalue in an image, or the movement in the image, for example. Next, theAF control section 340 determines whether or not a scene change has beendetected (S106). When the scene change has not been detected, theprocess in S105 is repeated. When the scene change has been detected,the standby operation is terminated. When the standby operation isterminated, the AF control section 340 resumes the focusing operation.While the standby operation is being performed, the AF control section340 maintains the focus lens position to be at the in-focus position,and does not drive the focus lens.

Next, a focusing operation (S101) performed by the AF control section340 is described in detail with reference to a flowchart in FIG. 6.

<AF Region Setting>

When the operation starts, the region setting section (AF region settingsection) 2010 first sets the AF region including a plurality of blockson the image (S201). FIG. 7 illustrates an example of the AF region thusset. In FIG. 7, an outer rectangle represents the entire image, and eachrectangle including the sign A represents an evaluation block that is atarget of calculating the AF evaluation value, the feature amount, orthe like as described later. In FIG. 7, a range surrounding all of theevaluation blocks serves as the AF region. In FIG. 7, total of 20evaluation blocks, including five blocks in a lateral direction and fourblocks in a vertical direction, are set to be in a center portion ofimage data.

<Block AF Evaluation Value Calculation>

The AF evaluation value calculation section 2030 calculates the block AFevaluation value (the AF evaluation value) of each evaluation block,based on a pixel value of the image data output from the pre-processingsection 320 (S202). The block AF evaluation value increases as thefocusing level of the subject in the block increases. For example, theblock AF evaluation value is calculated as a sum of output valuesobtained as a result of applying a bandpass filter to each pixel in animage in each evaluation block.

<Block Direction Determination>

The direction determination section 2040 determines the target in-focusdirection for each evaluation block from the block AF evaluation valueof the evaluation block (S203). An example of the determination methodis described with reference to FIG. 8A, with AfVal[N] representing thelatest block AF evaluation value (in the current frame) output from theAF evaluation value calculation section 2030, and AfVal[N−1] andAfVal[N−2] respectively representing the block AF evaluation valuesoutput in the previous frame and in the frame immediately before theprevious frame. The direction determination section 2040 calculates ablock AF evaluation value change amount α with the following Formula(3).

α={(AfVal[N]+AfVal[N−2])/2)}−AfVal[N−1]  (3)

Through this process, the in-focus direction can be accuratelycalculated with a shift operation performed together with the wobblingas illustrated in FIG. 8A (even when the amounts of the movement of thefocus lens in the NEAR direction and in the FAR direction are notconstant).

Comparison between the AF evaluation values obtained in the two framesin series with the focus lens moved as illustrated in FIG. 8A has thefollowing problem. Specifically, the focus lens moves by an amountcorresponding to the wobbling amount between N−2 and N−1, but moves byan amount as a result of adding a shifted amount to the wobbling amountbetween N−1 and N. Thus, the lens movement amount varies among timings,and thus the direction cannot be stably determined.

FIG. 8B illustrates a general wobbling operation as a comparativeexample. In this case, the movement direction of the wobbling centerposition is determined by determining the target in-focus direction,with the AF evaluation values calculated in two frames (N−2 and N−1) andcompared with each other in the Nth frame. In this process, a constantmovement amount (amplitude) of the focus lens can be achieved for eachdirection determination process. Unfortunately, the method in FIG. 8Bcan only obtain a single direction determination result per threeframes, and is difficult to achieve a high speed focusing operation andthe like. For example, the direction determination result can beobtained in only two frames (N and N+3) within the range illustrated inFIG. 8B.

In view of this, Formula (3) described above is used so that asubstantially stable lens movement amount can be achieved. For example,in the Nth frame, determination is performed for the movement of thefocus lens between an average position between the Nth and the N−2thframes and the position in the N−1 th frame. In the subsequent frame(N+1), the determination is performed for the movement of the focus lensbetween an average position between N+1 th and N−1 th frames and theposition in the Nth frame. The amount of this movement is substantiallythe same as the movement amount in the Nth frame. The same applies tothe subsequent frames.

With this configuration, the wobbling and the shift operation can beperformed at the same time as illustrated in FIG. 8A, whereby thesubject can be quickly brought into focus. Furthermore, the directiondetermination result can be obtained in each frame.

The value of the block AF evaluation value change amount α, obtainedwith Formula (3) described above, varies not only based on the blurringlevel, but also varies based on the luminance or the contrast of thesubject. In this example, an index value representing the blurring levelin each evaluation block is acquired, and thus components based on theluminance and the contrast of the subject are preferably removed. In thepresent embodiment, the block AF evaluation value change amount αobtained by Formula (3) described above is normalized to obtain a blockAF evaluation value change rate β. Specifically, the following Formulae(4) and (5) may be used. In the present embodiment, the determinationresult is determined to be NEAR when the block AF evaluation valuechange rate β is a positive value, and is determined to be FAR when theblock AF evaluation value change rate β is a negative value. Thus, whenAfVal[N] is calculated from an image as a result of the movement of thefocus lens in the NEAR direction, the following Formula (4) is used.When AfVal[N] is calculated from an image as a result of the movement ofthe focus lens in the FAR direction, the following Formula (5) is used.In the formulae, Ave(a, b, c) represents an average value of a, b, andc.

β=α/Ave(AfVal[N],AfVal[N−1],AfVal[N−2])  (4)

β=−1*α/Ave(AfVal[N],AfVal[N−1],AfVal[N−2])  (5)

The block AF evaluation value change rate β obtained with Formula (4) or(5) descried above is a value obtained by normalizing the block AFevaluation value change amount α. Thus, a substantially constant valueis obtained in accordance with the focusing level change range in thewobbling, regardless of the contrast and the brightness of the subject.

<Block Feature Amount Calculation>

The block feature amount calculation section 2020 calculates a featureamount of each evaluation block (such as color information, luminanceinformation, a size of a bright spot) based on image data output fromthe pre-processing section 320 (S204). The calculation of the featureamount is implemented with a widely known method which will not bedescribed in detail.

<Invalid Block Setting>

The invalid block setting section 2050 sets the invalid block based onthe block AF evaluation value change rate β obtained in S203 and theblock feature amount obtained in S204 (S205).

First of all, the invalid block setting section 2050 sets an evaluationblock with an absolute value of the block AF evaluation value changerate β being outside a predetermined range to be an invalid block. Forexample, the evaluation block satisfying |β|<first threshold or|β|>second threshold (>first threshold) is set to be an invalid block. Acorrect direction determination result cannot be obtained from a subjectwith an insufficient contrast or from a largely blurred image. In such acase, the block AF evaluation value change rate β is small.

A correct direction determination result cannot be obtained when asubject in the captured image changes due to the movement of the subjector the like, when the treatment tool suddenly enters the image, or whenany of images used for calculating the block AF evaluation value changerate β involves motion blur. In such cases, the block AF evaluationvalue change rate β is large.

In view of this, an evaluation block involving an excessively small orlarge block AF evaluation value change rate β can be set to be theinvalid block, whereby a block with an unreliable block directiondetermination result can be set to be the invalid block, and a highlyaccurate in-focus direction determination process can be achieved.

The invalid block setting section 2050 detects an evaluation block, ahigh luminance portion, and a dark portion occupied by an object otherthan the tissue such as the treatment tool (a silver color or a blackcolor) and a bright spot, and the like, from the block feature amount(such as color information, luminance information, and the size of thebright spot) of the evaluation block, and sets such an evaluation blockto be the invalid block. With such a process, the block including notissue or the block with an unreliable block direction determinationresult can be set to be the invalid block.

<Block State Determination>

Then, the block state determination section 2060 determines the blockstate based on the block direction determination result obtained in S203and the invalid block setting result obtained in S205 (S206).

First of all, when there is an invalid block in the current frame or thetwo previous frames, the block direction determination result isunreliable, and thus the block state of the current frame is set to beinvalid. This is because the block direction determination is performedusing the block AF evaluation values in the current frame and the twoprevious frames as described above with reference to Formula (3).

In the present embodiment, the block state is updated when the blockdirection determination result remains to be the same for a thresholdperiod or more. Otherwise, the block state of the previous frame ismaintained. Thus, the reliability of the block state can be improved andcan be prevented from frequently changing.

FIG. 9 is a flowchart illustrating a process performed by the blockstate determination section 2060. When this process starts, first ofall, it is determined whether or not there is an invalid block in thecurrent frame or in the two previous frames (S301). When a result of thedetermination in S301 is Yes, a continuity counter is set to be 0(S302), and the block state is set to be the invalid block (S303). Thecontinuity counter represents the number of times the same directiondetermination result was obtained. When there is an invalid block in thetwo previous frames, the direction determination result (NEAR or FAR)for the current frame is unreliable. Thus, the continuity counterremains to be 0 regardless of the direction determination result in thecurrent frame in S302.

When a result of the determination in S301 is No, whether or not thedirection determination result is different between the current frameand the previous frame (frame one before the current frame) isdetermined (S304). When the result of the determination in S304 is Yes,it means that the direction determination result has changed. Thus, thecontinuity counter is set to be 1 because the same directiondetermination result is not maintained (S305). Furthermore, the blockstate in the previous frame is maintained because the value of thecontinuity counter does not exceed the threshold (S306).

When the result of the determination in S304 is No, there is no invalidblock in the two previous frames, and thus the same directiondetermination result is maintained. Thus, the process of determiningwhether the value of the continuity counter is smaller than thethreshold (for example, 2) is performed (S307). When the result of thedetermination in S307 is Yes, the same direction determination result ismaintained, and thus the continuity counter is incremented (S308).However, the block state in the previous frame is maintained because thedirection determination result is not maintained long enough (S309).

When the result of the determination in S307 is No, the same directiondetermination result is maintained long enough. Thus, the block state ischanged due to the direction determination result in the current frame(S310).

FIG. 10 illustrates a specific example. In FIG. 10, a given evaluationblock in a plurality of evaluation blocks is set as a target. Thedirection determination result in S203 and the invalid block settingresult in S205 are illustrated in the upper stage, and the block stateis illustrated in the lower stage.

The target evaluation block is set to be the invalid block in a frame A1in FIG. 10, and the block state in the frame is also set to be invalid.The direction determination result is FAR in a frame A2, and is NEAR ina frame A3. Still, the block states in these frames are invalid blockbecause there is an invalid block in the two previous frames.

In a frame A4, the direction determination result is switched to FARfrom NEAR, but the continuity counter is 1 because the result FAR is notmaintained. In this example, the threshold is 2, and thus the continuitycounter≤threshold is satisfied, whereby the previous frame is maintainedin the frame A4. Thus, the block state NEAR is maintained, even when thedirection determination result is FAR.

The same applies to a frame A5. There is no invalid block in the twoprevious frames from the frame A5. The continuity counter in the frameA5 is 1 because the continuity counter in the previous frame A3 is 0.Thus, continuity counter≤threshold is satisfied, whereby the invalidblock from the previous frame (A3) is maintained in the frame A5.

<Invalid Frame Setting>

Next, the invalid frame setting section 2070 sets the invalid frame(S207). The image in the frame determined to be an invalid frame is notsuitable for determining the in-focus direction, and results in thedirection determination result with a low reliability. Thus, thein-focus object plane position is not moved based on such a directiondetermination result. Specifically, only the wobbling operation isperformed, with the focus lens moved by an amount corresponding to thewobbling amount without moving the center position (without moving thefocus lens by an amount corresponding to the shift amount) for thewobbling operation. The invalid frame is set when mist is detected, orwhen an invalid subject is detected.

Thus, the invalid frame setting section 2070 first detects the mistbased on the direction determination result for each evaluation blockoutput from the direction determination section 2040. As describedlater, the mist is detected while taking the continuity of the directiondetermination result among frames into consideration, and thus the blockstate output from the block state determination section 2060 is notinvolved in the detection.

The mist produced during the endoscopic procedure might lead to afailure to accurately determine the block detection, and thus mightresult in an unstable focusing operation. The mist is produced only whenthe user uses the treatment tool such as an electrocauter to perform atreatment. Thus, the subject is somewhat in focus while the mist isbeing produced. Thus, the focusing operation may be interrupted when themist is detected, so that the treatment performed by the user can beprevented from being affected.

The mist is detected based on the level of variation of the directiondetermination result among frames, and among evaluation blocks in aframe. When the mist is produced, the density of the mist on the imagelargely fluctuates, and the distribution of the variation largelychanges within a short period of time. As a result, the AF evaluationvalue similarly changes largely. Thus, the block direction determinationresult makes a large temporal and special change in FIG. 11. Thus, themist can be detected with the method described above.

The level of variation is detected as follows for example. First of all,whether or not the direction determination result is different from thatin the previous frame is determined for each evaluation block (temporalvariation level determination). Then, the direction determination resultin the current frame of the evaluation block, with the directiondetermination result changed from that in the previous frame, iscompared with those in peripheral evaluation blocks. The number ofevaluation blocks with the different direction determination result iscounted (spatial variation level determination). A block the totalnumber of counted blocks of which has exceeded a threshold is determinedto be a mist block. Thus, the mist block is an evaluation block that isdetermined to have a large temporal variation level and a large spatialvariation level.

When the number of mist blocks exceeds a predetermined threshold, theinvalid frame setting section 2070 determines that the mist is detectedand sets the current frame to be the invalid frame.

The invalid frame setting section 2070 may use a motion vector toperform the mist detection. For example, a motion vector representingthe movement between two different images acquired at different timings(specifically, images in two frames in series) is obtained from theimages. This motion vector is obtained for each of a plurality of points(regions) set on the image. Thus, a plurality of motion vectors arecalculated from a process using the images in the two frames in series.The motion vector is susceptible to the mist, and thus a specialvariation is large among the plurality of motion vectors obtained whenthe mist is produced.

Thus, the invalid frame setting section 2070 determines the reliabilityof the motion vector based on a spatial similarity among the motionvectors, to perform the mist detection. The motion vectors with a highspatial similarity are calculated based on a signal component and arenot calculated based on a noise component, and thus are determined to be“reliable”.

Specifically, first of all, one motion vector (hereinafter, referred toas a target motion vector) is selected from a plurality of local motionvectors in the image. Then, whether or not a motion vector adjacent tothe target motion vector thus selected is a similar vector isdetermined, based on a difference between the target motion vector andthe adjacent motion vector. This determination process is performed onall of the adjacent motion vectors. Then, the number of similar vectorsis counted to be compared with a predetermined threshold. The targetmotion vector with the number of similar vectors exceeding the thresholdhas spatial similarity with the peripheral motion vectors, and thus isdetermined to be “reliable”. The target motion vector with the number ofsimilar vectors not exceeding the threshold is determined to be“unreliable”. This determination is performed on all of the motionvectors in the image, whereby whether or not each of the motion vectorsis reliable is determined. The reliability of the entire image isdetermined based on the reliabilities of the motion vectors, and themist is determined to be detected for the image with a low reliability.For example, the mist may be determined to have been detected when thenumber or ratio of the motion vectors with a low reliability exceeds agiven threshold.

The invalid frame setting section 2070 may perform the mist detection byusing both the direction determination result and the motion vectors.For example, the mist may finally be determined to have been detected,when the mist is detected with both of the direction determinationresult and the motion vectors. Thus, the mist detection can be performedwith a higher accuracy.

The invalid frame setting section 2070 may set a target frame to be theinvalid frame when an invalid subject is detected. Specifically, theinvalid subject detection is performed based on the block state outputfrom the block state determination section 2060. When the AF region islargely occupied by the invalid blocks (the treatment tool, a brightspot, a high luminance portion, a dark portion, and a block with theblock AF evaluation value change rate (3 outside the predeterminedrange), the focusing operation cannot be accurately performed. Thus,when the number of invalid blocks set in S205 exceeds a predeterminedthreshold, it is determined that there is an invalid subject, and thetarget frame is set to be the invalid frame.

A scene resulting in the invalid frame set should not be maintained fora long period of time during an endoscopic procedure. Thus, the impactof the interruption of the focusing operation on the treatment by theuser is limited.

The focus control section 2000 determines whether or not the targetframe has been set to be the invalid frame by the invalid frame settingsection 2070 (S208). When the target frame has been set to be theinvalid frame, the focus lens position is determined (S211) with theprocesses such as the in-focus direction determination process (S209)and the in-focus determination (S210) skipped. The processes in S209 toS211 are sequentially performed when the target frame is not set to bethe invalid frame.

As described above, the focus control section 2000 (processor) accordingto the present embodiment further includes the invalid frame settingsection 2070 that sets the invalid frame based on at least one of thedirection determination result for each of a plurality of regions and afeature amount of the plurality of pixels in the regions. The focuscontrol section 2000 does not move the in-focus object plane positionbased on the direction determination result in the frame determined tobe the invalid frame.

With this configuration, the in-focus object plane position can beprevented from being shifted in an inappropriate direction when thedirection determination result is unreliable. The “movement based on thedirection determination result” represents the movement of the focuslens by an amount corresponding to the shift amount of the wobblingcenter position. Still, even when the frame is set to be the invalidframe, the wobbling operation (the movement of the focus lens by thewobbling amount from the wobbling center position) for determining thedirection cannot be prevented.

Thus, the invalid frame setting section 2070 (processor) sets theinvalid frame based on at least one of information on the spatialvariation (variation level) of the direction determination result,information on the temporal variation of the direction determinationresult, and information on the variation of the motion vector serving asa feature amount.

The information on the temporal variation represents the level ofvariation (level of change) of the direction determination result in agiven region over time as described above. The information on thespatial variation represents a level of variation between the directiondetermination result in a given region and the direction determinationresult in its periphery (four blocks on the upper, lower, left and rightsides, eight surrounding blocks, or the like in the example of theevaluation blocks in FIG. 7 for example). The information on thevariation of the motion vector represents the level of variation amongthe plurality of motion vectors set on a single image.

With this configuration, the in-focus direction determination process orthe like can be skipped when the mist is produced or when an image of aninvalid subject is captured. Thus, a risk of moving the focus lens in awrong direction can be reduced. Any one of the three types of variationinformation or a combination of any two of the three types of variationinformation may be used. For example, the block AF evaluation valuechange rate β is small around the in-focus position, resulting in adirection determination result with a large temporal and spatialvariation level. Thus, a frame might be erroneously determined to be theinvalid frame, even when the invalid frame needs not to be set. Thelevel of variation among motion vectors is small also around thein-focus position. Thus, the invalid frame can be accurately set bycombining the variation information on the direction determinationresult with the variation information on the motion vector.

<In-Focus Direction Determination>

The in-focus direction determination section 2080 determines the finalin-focus direction, based on the block state of each evaluation blockoutput from the block state determination section 2060 in S206 (S209).

As described above, the focus control section 2000 (processor) accordingto the present embodiment includes the invalid region setting section(invalid block setting section 2050) that sets the invalid region, basedon the AF evaluation value of each of the plurality of regions or afeature amount of the plurality of pixels in the regions. Thus, thefocus control section 2000 (in-focus direction determination section2080) determines that the in-focus direction is toward the NEAR side,when a ratio (nearRatio) of NEAR area information to area information onvalid regions that are a plurality of regions excluding at least theinvalid region is larger than a predetermined threshold (TH_NEAR)corresponding to the weight information.

FIG. 12 is a flowchart illustrating a process performed by the in-focusdirection determination section 2080. When this process starts, first ofall, the number of valid blocks in the AF region is counted (S401). Thevalid block is an evaluation block that is not the invalid block. Whenthe block state includes three states of NEAR, FAR, and invalid, thenumber of valid blocks is the sum of the numbers of the NEAR blocks andthe FAR blocks.

Next, the number of blocks with the block state NEAR is counted (S402),the ratio (nearRatio) of the NEAR blocks to the valid blocks iscalculated (S403). Then, whether or not the nearRatio exceeds thethreshold TH_NEAR is determined (S404), and the in-focus direction isdetermined based on a result of this determination. This corresponds toa process of determining the in-focus direction through the weightedcomparison between the NEAR blocks and the FAR blocks described above.

Specifically, the in-focus direction is determined to be NEAR whennearRatio>TH_NEAR is satisfied (S405) and is determined to be FAR whennearRatio≤TH_NEAR is satisfied (S406).

With this process, for example, the tissue can be accurately broughtinto focus even when a part of the treatment tool is failed to bedetected as the invalid block and thus is detected as the valid block inthe invalid block setting process in S205. This is because even when apart of the treatment tool is set to be the valid block to have theblock state (direction) different from the block state (direction) ofthe tissue, the in-focus direction is determined with the blocks stateof the tissue because the number of the block corresponding to thetissue is larger in the AF region as a whole.

When the subject has a depth as illustrated in FIG. 13, in the process,a larger value of TH_NEAR leads to the final in-focus position set to bemore on the deep side (farther from the imaging section 200). This isbecause when the value of TH_NEAR is large, the in-focus direction isnot determined to be NEAR and the in-focus object plane position ismoved toward the FAR side unless the ratio of the NEAR block isconsiderably high.

Thus, for example, the user can adjust the in-focus position as desiredby adjusting the value of TH_NEAR.

Alternatively, the focus control section 2000 (processor) according tothe present embodiment further includes a target distance estimationsection (not illustrated in FIG. 4 and the like) that estimates arelative distance between the subject determined to be the target of theuser and the imaging section 200, based on the image. Thus, the focuscontrol section 2000 may change the threshold based on an estimationresult by the target distance estimation section. For example, the focuscontrol section 2000 (in-focus direction determination section 2080) mayautomatically adjust the value of TH_NEAR based on a result ofestimation of the state of the subject from a luminance distribution ofthe entire image by the target distance estimation section.

Specifically, the target of the user can be estimated to be an organposition on a closer side in an abdominal cavity when an image with abright center portion and a dark peripheral portion as illustrated inFIG. 14 is used. In such a case, the target organ positioned on thecloser side can be accurately brought into focus, with the value ofTH_NEAR set to be small.

Thus, the focus control section 2000 (processor) according to thepresent embodiment performs comparison between the NEAR area informationand the FAR area information that have been weighted by the secondweight information, as the weighted comparison. With this configuration,the in-focus position (the in-focus object plane position with thetarget subject determined to be in focus, or the focus lens positionachieving the in-focus object plane position) can be flexibly set.

The method according to the present embodiment described above uses thesecond weight information. Note that the first weight information mayalso be used in the present embodiment.

The focus control section 2000 (processor) according to the presentembodiment further includes the target region estimation section (notillustrated in FIG. 4 and the like) that estimates a region determinedto be a target of the user, based on the image. The focus controlsection 2000 may set weight information (first weight information)achieving a large weight on the region estimated by the target regionestimation section, and may calculate the NEAR area information based onthe weight information set.

For example, when the user performs a treatment on a target as a part ofa subject with a depth as illustrated in FIG. 15, the target region ofthe user needs to be accurately brought into focus. In this case, forexample, the target region estimation section first estimates the targetblock corresponding to the target region as illustrated in FIG. 15, andthe focus control section 2000 (in-focus direction determination section2080) counts the weight N (the first weight information, N>1) of thetarget block. With this process, the weight on the state (direction) ofthe target block increases in the calculation of the nearRatio. As aresult, the weight of the state of the target block also increases fordetermining the in-focus direction, whereby the target region can beaccurately brought into focus.

Thus, the focus control section 2000 (processor) according to thepresent embodiment obtains the NEAR area information based on the firstweight information. Specifically, the NEAR area information is obtainedbased on the first weight information set to the region determined to beNEAR. As described above, when all of the plurality of regions(evaluation blocks) have the same area, the sum of the weights of theregions determined to be NEAR serves as the NEAR area information. TheNEAR area information can be more generalized, that is, may be the sumof the products of the areas and the weight information of the regions.With this configuration, the target region in the image can beappropriately brought into focus.

The process described above is performed based on the region determinedto be NEAR. Thus, the focus control section 2000 according to thepresent embodiment calculates the NEAR area information based on thefirst weight information. Alternatively, the focus control section 2000may perform the process based on the region determined to be FAR. Insuch a case, the focus control section 2000 obtains FAR area informationbased on the first weight information set to the region determined to beFAR. Specifically, the focus control section 2000 sets weightinformation (first weight information) achieving a large weight on theregion estimated by the target region estimation section, and calculatesthe FAR area information based on the weight information set.

The target block may be estimated as follows for example. The regionwhere the user performs the treatment involves a large variation of thetreatment tool and the tissue. Thus, the AF evaluation value and theblock feature amount (color information, luminance information) of theevaluation block largely change within a short period of time. Thus, thetarget block may be estimated from an amount of change of these valuesover time. A motion vector of the evaluation block may be calculated byusing a known method, and the target block may be estimated based on theamount of change in the magnitude and the direction of the motion vectorover time. The block with a large amount of change as well as itsperipheral blocks may be estimated to be the target block so that aratio of the invalid block (treatment tool) in the target block can beprevented from being large.

FIG. 16 is a flowchart illustrating a process using the first weightinformation. When this process starts, first of all, the target block isestimated (S501). Then, the valid blocks are counted (S502). Note thatS502 is different from S401 in that the weight N is used for thecounting. Specifically, the blocks are counted with the weight of thetarget block being N and the weight of a block other than the targetblock being 1. The number of NEAR blocks are counted in a similarmanner, that is, counted with the weight of the target block being N andthe weight of a block other than the target block being 1 (S503).

The processes in S504 to S507, after the counting, are respectively thesame as those in S403 to S406 in FIG. 12.

<In-Focus Determination>

The in-focus determination section 2090 determines whether or not thefocus lens has reached the in-focus position based on the in-focusdirection (NEAR/FAR) output from the in-focus direction determinationsection 2080 and a position where the in-focus direction has beenreversed (S210).

The focus lens that has reached the in-focus position makes areciprocating motion relative to the in-focus position as illustrated inFIG. 17. This is because the in-focus direction is not reversed unlessthat lens passes through the in-focus position by a certain distance dueto an extremely small value of the block AF evaluation value change rateβ at the in-focus position. In the in-focus state, the in-focusdirection is always reversed at substantially the same position. Thus,whether or not the in-focus state is achieved can be determined bydetermining whether or not the following two conditions are satisfied:(1) the reciprocating motion is performed for a predetermined number oftimes; and (2) the variation of the reversing position of thereciprocating motion is small.

FIG. 18 is a flowchart illustrating a process performed by the in-focusdetermination section 2090. When this in-focus determination processstarts, first of all, whether or not the in-focus direction has beenreversed is determined (S601). When a result of the determination inS601 is No, the in-focus determination process in the current frame isterminated.

When a result of the determination in S601 is Yes, a value of thereversal counter is determined (S602). The reversal counter is a counterindicating the number of times the in-focus direction has been reversed.When the value of the reversal counter is 0, the wobbling centerposition at the time when the in-focus direction is reversed is storedin the memory 1 (S603), and the reversal counter is incremented (S604)to be 1 at this timing. In FIG. 17, the focus lens position (wobblingcenter position) at B1 is stored in the memory 1.

When the value of the reversal counter is 1, the wobbling centerposition at the time when the in-focus direction is reversed is storedin the memory 2 (S605), and the reversal counter is incremented (S606)to be 2 at this timing. In FIG. 17, the focus lens position (wobblingcenter position) at B2 is stored in the memory 2.

Through the processes in S603 and S605, reference positions on the FARside and the NEAR side in the reciprocating motion are stored in thememories. Whether the memory 1 corresponds to NEAR or FAR depends on thesituation. In the reversal detection thereafter, whether or not thevariation between the wobbling center position at the time of thedetection and the reference position stored in the memory is small maybe determined.

Specifically, in S602, when the value of the counter is 2 or more, thewobbling center position at the time when the in-focus direction isreversed is compared with the position (the value stored in the memory 1or the memory 2) at the time when the in-focus direction is reversed bythe reversing immediately before the previous reversing, and whether ornot a resultant difference (absolute value) is equal to or smaller thana threshold is determined (S607). The position at the time when thein-focus direction is reversed by the reversing immediately before theprevious reversing is used as a comparison target because the reversingfrom FAR to NEAR and reversing from NEAR to FAR are alternately detectedas illustrated in FIG. 17. For example, information obtained at B3 iscompared with the information obtained at B1, that is, the informationin the memory 1, and information obtained at B4 is compared with theinformation obtained at B2, that is, the information in the memory 2.

When the resultant difference is equal to or small than the threshold(Yes in S607), the information, in the memory 1 or the memory 2, used inthe comparison is updated with the wobbling center position in thecurrent frame (S608), and the reversal counter is incremented (S609). Atthe timing B3, the information in the memory 1 is updated with thewobbling center position at this timing, and at the timing B4, theinformation in the memory 2 is updated with the wobbling center positionat this timing. The same applies to the subsequent timings. When thedifference is larger than the threshold (No in S607), the in-focus stateis determined to be not achieved and the counter is set to 0 (S610).

After the process in S609, whether or not the reversal counter hasexceeded a focusing completion determination threshold is determined(S611). When a result of the determination in S611 is Yes, the in-focusstate is determined to be achieved (S612). In the example illustrated inFIG. 17, the reversal counter>5 is set as the condition, and thus theresult of the determination in S612 is Yes at the timing B6 at which thereversing occurs for the sixth time.

As described above, the focus control section 2000 (the processor, thein-focus determination section 2090 in a narrow sense) stops thefocusing operation when the count (reversal counter) indicating how manytimes switching of the movement of the in-focus object plane positionfrom the NEAR to the FAR or from the FAR to the NEAR has occurredexceeds a predetermined switching threshold (the focusing completiondetermination threshold). This corresponds to the determination in S611described above. In this process, the focus control section 2000 resetsthe count indicating how many times the switching has occurred, when avariation of the in-focus object plane position corresponding toswitching from the NEAR to the FAR or a variation of the in-focus objectplane position corresponding to switching from the FAR to the NEARexceeds a predetermined variation threshold. This corresponds to theprocesses in S607 and S610 described above. The relationship between thein-focus object plane position and the focus lens position is likely tobe recognized when the focus control device (endoscope apparatus) isdesigned. Thus, the in-focus object plane position can also be regardedas the focus lens position.

With this configuration, the in-focus determination can be performedbased on whether or not the reciprocating motion illustrated in FIG. 17has occurred.

<Focus Lens Position Determination>

The focus lens position determination section 2095 determines the nextfocus lens position by using the setting result obtained by the invalidframe setting section 2070, the in-focus direction determined by thein-focus direction determination section 2080, and the determinationresult obtained by the in-focus determination section 2090 (S211).

FIG. 19 is a flowchart illustrating a process performed by the focuslens position determination section 2095. When this process starts,first of all, whether or not the current frame is the invalid frame isdetermined (S701). When the current frame is the invalid frame, thefocus lens position is determined to maintain the current wobblingcenter position (S702). Specifically, the shift amount is set to be 0,and the current wobbling operation is maintained.

When the result is No in S701, whether or not the subject has beendetermined to be in-focus in S210 is determined (S703). When the subjecthas been determined to be in-focus, the focus lens position is set to bethe average position of the values stored in the memories 1 and 2(S704). At the timing B6 when the subject is determined to be in-focusin FIG. 17, the wobbling center position at the timing B5 is stored inthe memory 1 and the wobbling center position at the timing B6 is storedin the memory 2. Thus, the process in S704 corresponds to a process ofdetermining the focus lens position implementing the focus lens movementfor B7.

When the subject is determined to be out of focus, the in-focusdirection determined in S209 is determined (S705). When the in-focusdirection is NEAR, the focus lens position is determined so that thewobbling center position moves toward the NEAR side (so that thein-focus object plane position moves toward the imaging section 200) bythe shift amount (S706). When the in-focus direction is FAR, the focuslens position is determined so that the wobbling center position movestoward the FAR side (so that the in-focus object plane position movesaway from the imaging section 200) by the shift amount (S707).

4. Modifications

The method according to the present embodiment is not limited to thatdescribed above, and various modifications may be made. Modifications ofthe processes performed by the sections of the AF control section 340are described below.

<Block AF Evaluation Value and Direction Determination>

The AF evaluation value calculation section 2030 is not limited to thecalculation of a single block AF evaluation value for a single block,and may calculate a plurality of block AF evaluation values using aplurality of band pass filters with different frequency bands. Thedirection determination section 2040 may obtain the block AF evaluationvalue change rate β from each of the plurality of block AF evaluationvalues, and may determine the block direction based on the block AFevaluation value change rate β. Thus, the block direction determinationcan be accurately performed for a subject with various frequency bands.For example, the direction determination section 2040 obtains thedirection determination result from a plurality of block AF evaluationvalue change rates β, and may prioritize a result obtained from afrequency band that is expected to be a target of the user, whendifferent results are obtained.

<Invalid Block Setting and Block State Determination>

The value (absolute value) of the block AF evaluation value change rate(3 is extremely small around the in-focus position. Thus, the blockstate of most of the blocks might be determined to be invalid, resultingin a failed focusing operation. To address this, a modified process maybe performed with a block with the block AF evaluation value change rateβ not exceeding a threshold (low contrast) not be set as the invalidblock.

Specifically, the invalid block setting section 2050 sets an evaluationblock with an excessively high block AF evaluation value change rate (3,an evaluation block occupied by a treatment tool (silver color or blockcolor), a bright spot, and the like, and an evaluation blockcorresponding to a high luminance portion, a dark portion, or the liketo be the invalid block, and sets an evaluation block with anexcessively low block AF evaluation value change rate β to be a lowcontrast block.

When the current frame is the low contrast block, the block statedetermination section 2060 maintains the block state of the previousframe, instead of setting the block state to be invalid. FIG. 20illustrates a specific example where the subject is determined to be inthe low contrast state and the continuity counter is reset to 0 inframes C1 and C2. Still, the block state in the previous frame ismaintained in these frames (the block state FAR in a frame C3 in FIG.20). In a portion around the in-focus position, the absolute value ofthe block AF evaluation value change rate β increases again when thefocus lens passes through the in-focus position. Thus, the low contraststate does not continue for a long period of time. For example, a resultother than the low contrast is acquired as in a frame C4 in FIG. 20, andthe reciprocating motion around the in-focus position continues.

Thus, a state where the low contrast state continues for a long periodof time can be determined to be different from a state where the blockAF evaluation value change rate β temporarily decreases around thein-focus position. For example, this state corresponds to a state wherea blurring level is too high (largely blurred state) and a state wherethe subject is in the low contrast state, in which the direction cannotbe determined by the wobbling operation. Thus, the number of times theprevious block state was maintained (or the number of times the lowcontrast state was maintained) is counted, and the subject may bedetermined to be in the low contrast subject and the block state may beset to be invalid when a result of the counting exceeds a threshold.

<Invalid Frame Setting>

In the example described above, the invalid frame setting section 2070obtains a motion vector based on an image. However, this should not beconstrued in a limiting sense, and sensor information from a motionsensor may be used. The motion sensor is a sensor that detects a motionof the imaging section 200, such as an accelerometer or a gyro sensor.

In the above description, the invalid frame setting section 2070 employsa method based on the mist detection or the invalid subject detection.Note that the invalid frame may be set through other methods.Specifically, an accurate focusing operation is difficult to achieve dueto motion blur while the imaging section 200 is moving. Thus, theinvalid frame setting section 2070 performing the above described methodmay further detect the motion of the imaging section 200 and set a frameinvolving the motion of the imaging section 200 to be the invalid frame.

Specifically, when the magnitude of the motion vector is larger than agiven threshold, the frame is set to be the invalid frame. For thismotion vector, information that is similar to that of the motion vectorused for the mist detection may be used. Note that a large motion vectorcorresponding to the treatment tool is obtained also when the treatmenttool is largely moving with no relative movement between the imagingsection 200 and the target subject (tissue). However, the motion blur isnot large in such a situation, and thus the frame needs not to be theinvalid frame. Thus, a local motion vector and a global motion vectormay be obtained as the motion vectors, and the invalid frame may be setbased on the global motion vector as one of the motion vectors.

<In-Focus Direction Determination>

In the above description, the in-focus direction determination section2080 determines the in-focus direction based on the block state in thecurrent frame. However, this should not be construed in a limitingsense. The in-focus direction determination section 2080 may perform aprocess of updating the in-focus direction when the same in-focusdirection is maintained for a plurality of times. With such a process,the stable focusing operation can be implemented with the in-focusdirection prevented from frequently changing.

<In-Focus Determination>

The condition used by the in-focus determination section 2090, includingthe small variation (change) in each of the position where the reversalfrom FAR to NEAR occurs and the position where the reversal from NEAR toFAR occurs, may further include other conditions. For example, avariation of the width of the reciprocating motion relative to thein-focus position is obviously not large. Thus, an excessively large orsmall difference between the position of the previous reversing and theposition of the latest reversing results in a low reliability. Thus, thein-focus state can be determined to be not achieved when a distance(difference, absolute value of the difference) between the positionwhere the reversing from FAR to NEAR has occurred and the position wherethe reversing from NEAR to FAR has occurred is outside a predeterminedrange.

<Focus Lens Position Determination>

The shift amount for moving the wobbling center position to the in-focusdirection may be a predetermined fixed value, or may be graduallyincreased when the same direction determination result is maintained.With such a process, a time required for the focus lens to reach thein-focus position can be shortened (high speed focusing operation can beachieved).

The shift amount may be reduced when the focus lens reaches the in-focusposition and starts the reciprocating motion. With such a process, asmall amplitude of the reciprocating motion can be achieved, wherebydeterioration of the image quality due to the reciprocating motionduring the in-focus determination can be reduced. The amplitude of thereciprocating motion corresponds to the difference between the positionof the reversing from FAR to NEAR and the position of the reversing fromNEAR to FAR described above in the modification of the in-focusdetermination section 2090. Thus, when the in-focus determinationsection 2090 determines whether or not the amplitude of thereciprocating motion is within a predetermined range, the “predeterminedrange” is preferably set while taking the shift amount intoconsideration.

Although only some embodiments of the present invention and themodifications thereof have been described in detail above, those skilledin the art will readily appreciate that many modifications are possiblein the embodiments without materially departing from the novel teachingsand advantages of the invention. Accordingly, all such modifications areintended to be included within the scope of the invention. For example,any term cited with a different term having a broader meaning or thesame meaning at least once in the specification and the drawings can bereplaced by the different term in any place in the specification and thedrawings. The configurations and the operations of the focus controldevice and the endoscope apparatus are not limited to those describedabove in connection with the embodiments. Various modifications andvariations may be made of those described above in connection with theembodiments.

What is claimed is:
 1. A focus control device comprising a processorincluding hardware, the processor being configured to implement: aregion setting process of setting a plurality of regions, each includinga plurality of pixels, to an image acquired by an imaging section; adirection determination process of obtaining a direction determinationresult for each region in some or all of the plurality of regions set,by determining whether a target focusing position that is a target of anin-focus object plane position is on a NEAR side or a FAR side relativeto a reference position; and focus control of determining an in-focusdirection by performing weighted comparison between NEAR areainformation, indicating an area of a region with the determinationresult NEAR, and FAR area information, indicating an area of a regionwith the determination result FAR, based on weight information and thedirection determination result, and of controlling the in-focus objectplane position based on the in-focus direction.
 2. The focus controldevice as defined in claim 1, the processor determining the in-focusdirection based on the weight information that is at least one of firstweight information indicating a level of contribution of each of theplurality of regions to the weighted comparison and second weightinformation indicating a level of contribution of the NEAR areainformation and the FAR area information to the weighted comparison. 3.The focus control device as defined in claim 2, the processor obtainingthe NEAR area information or the FAR area information based on the firstweight information.
 4. The focus control device as defined in claim 2,the processor performing comparison between the NEAR area informationand the FAR area information that have been weighted by the secondweight information, as the weighted comparison.
 5. The focus controldevice as defined in claim 1, the processor performing an invalid regionsetting process of setting an invalid region based on at least one of anAutoFocus evaluation value of each of the plurality of regions or afeature amount of the plurality of pixels in the regions, and theprocessor determining that the in-focus direction is toward a NEAR side,when a ratio of the NEAR area information to area information on validregions that are a plurality of regions excluding at least the invalidregion is larger than a predetermined threshold corresponding to theweight information.
 6. The focus control device as defined in claim 5,the processor performing a target distance estimation process ofestimating a relative distance between a subject determined to be atarget of a user and the imaging section, based on the image, and theprocessor changing the threshold based on an estimation result by thetarget distance estimation process.
 7. The focus control device asdefined in claim 5, the processor performing a target region estimationprocess of estimating a region determined to be a target of a user,based on the image, and the processor setting the weight informationachieving a large weight on the region estimated by the target regionestimation process, and calculating the NEAR area information or the FARarea information based on the weight information set.
 8. The focuscontrol device as defined in claim 1, the processor performing a processof setting an invalid frame based on at least one of the directiondetermination result for each of the plurality of regions and a featureamount of the plurality of pixels in the regions, and the processor notmoving the in-focus object plane position based on the directiondetermination result in a frame determined to be the invalid frame. 9.The focus control device as defined in claim 8, the processor settingthe invalid frame based on at least one of information on a spatialvariation of the direction determination result, information on atemporal variation of the direction determination result, andinformation on a variation of a motion vector serving as the featureamount.
 10. The focus control device as defined in claim 1, theprocessor stopping a focusing operation when a count indicating how manytimes switching of the movement of the in-focus object plane positionfrom the NEAR to the FAR or from the FAR to the NEAR has occurredexceeds a predetermined switching threshold.
 11. The focus controldevice as defined in claim 10, the processor resetting the countindicating how many times the switching has occurred, when a variationof the in-focus object plane position corresponding to switching fromthe NEAR to the FAR or a variation of the in-focus object plane positioncorresponding to switching from the FAR to the NEAR exceeds apredetermined variation threshold.
 12. An endoscope apparatus comprisingthe focus control device as defined in claim
 1. 13. A method foroperating a focus control device, the method comprising: setting aplurality of regions, each including a plurality of pixels, to an imageacquired by an imaging section; performing a direction determinationprocess of obtaining a direction determination result for each region insome or all of the plurality of regions set, by determining whether atarget focusing position that is a target of an in-focus object planeposition is on a NEAR side or a FAR side relative to a referenceposition; and determining an in-focus direction by performing weightedcomparison between NEAR area information, indicating an area of a regionwith the determination result NEAR, and FAR area information, indicatingan area of a region with the determination result FAR, based on weightinformation and the direction determination result, and controlling thein-focus object plane position based on the in-focus direction thusdetermined.